1. The Status Quo
Software developers have been trying since the commercial electronic computer was introduced in 1951 to make computers reliably support organizations in all areas of activity, such as finance, sales, distribution, manufacturing, personnel, purchasing, etc. However, the complexity of rules within an organization, their constant changeability, and the traditional methods of encoding or representing those rules have all resulted in systems that, while partly successful, are very large, complex, expensive, and time-consuming to build, and technically difficult (sometimes impossible) to change.
Traditional software systems are built by representing the rules of the organization in software, while the facts describing the environment are normally encoded as data (e.g. the quantity on hand of a particular product).
In traditional software systems, rules are specified using three programming language constructs: (1) “If-Then-Else” constructs that tell the application what to do if certain conditions are either true or not true, (2) “Case” statements that allow more complicated choices where any number of separate conditions can be specified, and for each condition different software (and therefore different rules) can be specified to be executed, and (3) rules can be specified in a simple sequence: do A, then B, then C, without any conditions specified. Any number of such rules arranged in an ordered list is called an “application”. Frequently, rules are triggered based on the user performing certain actions (“events”) such as opening a form or clicking on a button.
In other cases rules are triggered based on the current state of affairs represented to the system by means of facts. Facts represent the state of entities that the rules act upon, such as customers or products. Facts are often represented using relational data structures, which require each table to have a single column constituting the unique key of each record (“primary key”), and any number of other columns as long as they are each related somehow to the primary key and do not duplicate other columns in the table. In a typical business application there can be millions of records distributed throughout thousands of separate tables. Depending on the application language and DBMS, facts can also be represented using more complex structures including objects.
2. Analysis of the Problem
We use the term “Ultra-Structure” to designate a new technology for analyzing and representing any kind of rules as data rather than as software in any software application. A full description of an earlier version of this invention was published in the January 1995 issue of the Communications of the ACM. Prior to that it was published by a withdrawn EU patent application. It has also been described conceptually in an article in Volume 133-1/3 (1999) of Semiotica. All of those documents focused on the application of Ultra-Structure theory to work management within organizations.
Much of the problem with current applications is that their designers have failed to understand that “work generation and management” is the common denominator of all organizational activities including (but not limited to) businesses, government entities, and charitable organizations, or that work generation and management can be accomplished and described solely by rules. Current applications do not define and encapsulate the rules and the rule-management process (rules about rules) underlying those various work activities such that the actual rules to be followed are defined as data that is read, executed, and managed with a relatively small and simple set of software programs, and that can be changed directly by trained rule-makers within the organization without recourse to modifying the computer program code.
Since current applications typically embody most if not all rules in the application software, and each group of applications is designed separately based on the area or sub-area of organization activity it is intended to support, current applications are extremely complex with many millions of lines of code and thousands of interlinked tables.
When one wishes to change the rules, the application software and data structures themselves must often be changed, an activity which has great risk of creating unexpected side-effects. Changes further require communicating organizational needs from a rule-maker to a computer programming expert, an activity which is fraught with the risks of misunderstandings. Adding, changing or deleting rules is thus an expensive, time consuming process which prevents organizations from changing their rules as quickly as they would like to (for example, in response to reorganizations, work process reengineering projects, changes in law, or as the result of a merger or acquisition). In some cases the changes to software and data structures take a long time to implement, or cannot be implemented at all without a rewrite of the system, so the organization must employ expensive “workarounds” to circumvent the inertia of the application that is supposed to support the organization.
Another limitation of current applications is that they deal only with “explicit” workorders such as an order from a customer to the organization to ship a product to the customer. The application's response and the organization's response to such a request are then defined by rules that are encoded either in the application's software or externally to the application via standard operating procedure manuals or in worker's minds.
Current applications generally are not designed around the fact that each explicit workorder invariably connotes a plurality of “implicit” work, i.e. actions necessary to carry out the explicit workorder, such as “perform a credit check” or “get the item from inventory”. Indeed each “implicit” workorder can have its own implicit workorders, and in general any “stimulus workorder” can generate multiple “response workorders” to as many layers of depth as rule-makers wishes to define.
Current applications also generally do not recognize that much of the work of an organization consists in grouping workorders together in various ways, and linking workorders to one another, for various purposes. For example, an “order” is a collection of individual explicit requests, and an “invoice” is a collection of delivered orders. Entities such as “invoice” have their own attributes, but their primary contents are defined by the workorders they reference.
Further, whenever an explicit workorder is made for the organization to do something, the workers often need to know other things before or as they do the work that is implicit in an explicit workorder. There is a need therefore for a system wherein workers who are carrying out such work can have access to other information which they need to know in order to do the work as well and as quickly as possible (such as what and where inventory is available).
As a result, executives and end-users often feel hindered rather than supported by the computer applications they use. Applications are often viewed as impediments, and a gigantic source of inertia and inefficiency that prevents organizations from being able to quickly respond to marketplace or other demands. It is not unusual for medium and large organizations today to maintain 10-50 million lines of code, at a direct cost to them, for software alone, of 1% or more of their total revenue. This situation has been caused by an inadequate analysis of work management and a design approach that calls for defining rules almost exclusively via software.
3. Alternative Approaches
There are a large number of approaches that perform some of the desired functions for an organization, but these are different than the invention in the ways noted below.
3a. PERT/CPM Systems
Program Evaluation and Review Technique (“PERT”) and Critical Path Modeling (“CPM”) systems are planning tools that allow its users to sequence a series of specified tasks, estimate work times and assignments for each task, and to thereby compute various dates (e.g. projected completion date) and resource loadings. They are helpful when planning projects that involve many tasks, primarily in terms of allocating work among resources and predicting a completion date for the project if all the tasks specified occur in the order specified. The nature and sequencing of tasks is fully defined in advance and is not contingent upon the results of real work being done, i.e. PERT/CPM systems do not normally specify alternative tasks and pathways to be followed under other-than-expected actual conditions. They provide no notification to workers of what the steps are, and no decision support for the workers who are to carry out the required work steps. They are useful as planning tools but they are not work process managers.
3b. Parameterized Application Software Packages
Parameter-driven applications allow users to specify which rules, from a limited list, should be followed by the application. These options are typically encoded by parameters (data values) specified by rule-makers and loaded at startup or during runtime. Parameter-driven applications giving users more flexibility about specifying the rules to be executed. However, this approach requires application developers to imagine and implement many sets of alternative rules, thereby making the application far larger and harder to maintain over time. The rules are still in the software.
As a result, organizations using such packages frequently must either (a) change their rules to fit the options provided by the package, or (b) invest enormous time, effort and money in changing the software in the package to support the organization's existing rules.
3c. Document Control Systems
Document Control Systems provide a way to pass documents around within an organization. These are typically used along with electronic images of the documents to be worked on. The technology has very simple conditions that are used to determine work processes, so conditional paths are difficult if not impossible to define.
3d. Expert Systems
In great contrast to IT practices, the Expert Systems branch of the Artificial Intelligence (“AI”) community represents most rules outside the software of a system, in an area usually called the “rulebase”, and keeps data regarding facts about the outside world in a “factbase” (usually not a Database Management System, “DBMS”, but rather any physical table-on a disk drive). The unique characteristics of an expert system are separate memory areas for rules and facts (which are loaded at system startup), a rule-to-fact matching capability (implemented as a cycle through a set of rules), and selection and execution of one or more rules, resulting in some change to the fact memory.
Expert systems are usually architecturally composed of an expert system language interpreter (an expert system “shell”), one or more tables containing rules (the “rulebase”) structured in some declarative rule language specifying and encapsulating the logic of the system, and one or more fact sources (usually from tables). The shell knows only how to read and apply the rulebase rules; all procedural knowledge is specified in the rulebase, and all state-of-the-world knowledge is specified in the factbase.
Expert system rules have a “left hand side” and “right hand side,” and are called “production rules” or “productions”. The left hand side of the rule is composed of fact patterns to be matched, while the right hand side is the action that the system should take against the factbase. Any number of such production rules arranged in an ordered or unordered list is called a “production rule system”. Production rules may be encoded in several ways but are mostly encoded in “Object-Attribute-Value” (“OAV”) data structures, where each object has x number of attributes, and each attribute may have y number of values.
Facts represent the objects of interest that the rules act upon, and are also typically represented using OAV data structures. To describe these in relational database terms, the “object” roughly corresponds to a primary key column of a table; the “attribute” maps to the name of a non-primary-key column in a table; and a “value” maps to the cell value found at the intersection of the key value on a unique row and the attribute column. Depending on the rule language, facts can also be represented using more complex structures including objects.
Expert Systems typically run in cycles, where each rule that is fired modifies the factbase content, so that the next cycle of the system may fire a different rule than the last cycle. These systems take the facts in seriatim order, and do not generalize or otherwise modify them; as a result they mimic, as they were intended to do, the human cognitive processes associated with diagnostic or exploratory activities, rather than a flexible, user-changeable way to analyze complex stimuli in parallel, after various complex classificatory transformations, to generate complex predefined responses.
The expert system shell maintains two separate memories: rules are loaded into rule memory and facts are loaded into the working memory (asserted) of the expert system shell. The expert system shell is independent from rule or fact implementation details. Typically expert system execution involves matching facts against the left hand side of rules cyclically until the system terminates. At the end of each cycle typically one matched rule is executed against its matching facts. As a result, facts are consumed and produced during each cycle. Execution continues until no more facts can be matched or until an explicit halting rule is fired. A central problem with rule based expert systems is efficiently handling the many-to-many matching problem between rules and facts.
Modern expert system shells make use of built-in optimization algorithms to reduce what would normally be an exponentially increasing execution time ((# Rules) times ((# Facts) raised to the power of (#Left-Hand Side Patterns))) to an execution time that increases linearly based on the size of the factbase. The optimization algorithm leverages the fact that most rulebases are static and that only a small percentage of facts changes during each cycle. The optimization algorithms are usually structured as networks that keep track of rule-fact matches and efficiently update the network to reflect fact consumption and production. Most modern production rule systems, and thus most modern expert systems, are designed this way.
In spite of these advances, expert systems generally start having maintenance problems before they even reach one thousand rules in the rulebase. Real-world rule-based applications will almost certainly exceed 100,000 rules, i.e. two orders or more of magnitude greater than is currently supportable.